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History  of  the  “Bell-Ringing”  Initiative 


•  Beginning  in  1999,  failures  in  high-profile  acquisition 
reform  programs  began  to  emerge  at  SMC 

❖  Performance  deficiencies,  extensive  software  defects 

❖  Large,  unanticipated  cost  and  schedule  overruns 

•  Extent  and  severity  of  software-related  problems  were 
not  understood  until  late  in  the  development  life  cycle 

•  One  SMC  System  Program  Director  expressed  the 
desire  to  have  a  set  of  “bell-ringing  criteria” 

❖  Early  warning  indicators  of  actual  or  potential  problems  in  one  or 
more  factors  that  influence  success  of  software-intensive  system 
acquisitions 

❖  Signal  the  need  for  additional  management  attention  to  the 
influencing  factors 
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Aerospace  “Bell-Ringing”  Research 


*  Aerospace  began  a  research  project  to  develop  a 
comprehensive  set  of  “bell-ringing  criteria” 

*  Factors  adversely  influencing  the  success  of  software¬ 
intensive  system  acquisition  were  determined  to 
extend  far  beyond  those  controlled  by  the  contractor’s 
software  development  teams 

*  Research  project  first  phase:  Define  a  Framework  for 
identifying  and  classifying  these  factors 

❖  Framework  to  define  the  relationships  and  interactions  of  the  factors 

❖  Framework  to  include  quantitative  indicators  derived  from  the 
factors 

-  These  indicators  to  provide  foundation  for  satisfying  the  realistic 
information  needs  of  the  acquisition  manager 

THE  AEROSPACE 

CORPORATION  SEslinger-5 


The  Approach  for  Defining  the  Framework  for 
Software  Acquisition  Early  Warning  Indicators 


Problem: 

•  Needed  a  simple  way  to  model  systemic 
influences  and  factors 

•  Needed  to  be  able  to  express  Cause 
(influence)  and  Effect  (factors) 

•  Needed  an  intuitive  information 
presentation  method 

Approach: 

•  Ishikawa  Diagrams  selected  for 

•  Ease  of  use 

•  Automated  tools 

•  Low  training  needs 

Example: 

Definitions: 

Factor 

•  Problem  Statement  -  A  statement  of  the 
problem — usually  provided  in  terms  of  a 

Sub-factor  \ 

necessary  improvement 

/7W\ 

•  Factor  -  Possible  causes  of  the  problem 

V  Problem 

•  Sub-factor  -  The  lower  level  factors  of  the 

Statement 

subset  problem 

•  Influence  -  Measurable  feature  of  the 
factor  or  sub-factor 
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The  Framework  for  Acquisition 
Early  Warning  Indicators 


1.  Acquisition  Environment  -  Influences  from  acquisition-related 
organizations  external  to  the  SPO,  including  acquisition 
management,  user/operator,  independent  test  and  certification, 
interfacing  system,  support,  and  Government  provider  (e.g., 

GOTS,  GFE/GFI,  facilities)  organizations 

2.  Acquisition  Program  Office  -  Influences  from  the  SPO  itself, 
including  everyone  from  the  SPO  director  down  through  the 
lowest  level  personnel,  plus  Aerospace,  other  FFRDCs  and  SETA 
contractors  that  support  the  SPO 

3.  Contractor  Environment  -  Influences  from  contractor  team 
organizations  external  to  contractor  team’s  engineering 
organizations,  including  corporate  and  program  management, 
quality  assurance/software  quality  assurance,  contracts, 
scheduling,  and  organizational  process  improvement  groups 

4.  Product  Development  -  Influences  from  contractor  team 
engineering  organizations,  including  systems  engineering, 
software  engineering,  specialty  engineering,  test  organizations, 
and  configuration  management 

5.  Unification  -  Influences  from  the  interrelationships  among  the 
first  four  factors,  such  as  customer  satisfaction,  communication 
and  agreements 
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The  Unique  Role  of  the  SPO 
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Example  Products  of  the 
Acquisition  Program  Office 
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System  Requirements 

♦>  Draft  System  Specification  (DSS),  Technical  Requirements 
Document  (TRD),  System  Performance  Specification  (SPS) 

Interface  Requirements 

❖  Interface  Requirements  Specification  (IRS),  Interface  Control 
Document  (ICD) 

Acquisition  Products 

❖  Request  for  Proposal  (RFP)  Package 

❖  Acquisition  Strategy  Briefing 

❖  Program  Milestone/Key  Decision  Point  (KDP)  Briefing, 
Integrated  Program  Summary  (IPS) 

Test  and  Evaluation  (T&E)  Products 

♦>  Test  and  Evaluation  Master  Plan  (TEMP) 

Cost  and  Schedule  Products 

❖  Cost  Analysis  Requirements  Document  (CARD) 

❖  Program  Schedule 
Other  Products 

❖  Trade  Studies  (e.g.,  Architecture  Studies) 

❖  Management  Reports 

❖  Processes,  Training 


S.  Eslinger  -  9 


Software  Measurement  and 
Acquisition  Management 


Management  Measurement 


Acquirer 


Contractor 


CDRL 

Requirements 


CDRL 

Items 


Information 

N>jy|eeds^ 


Development 

Management 

Process 


Acquisition 
Management 
Process 


Information 

Products 


Information 

Products 


Information 
Needs 


Acquisition 
Measurement 
Process 


Development 
Measurement 
Process 
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Acquisition  Program  Office 
Subfactors  and  Influences 


2.3 

Size 

and  Stability 
Sub-Factor 


2.2 

Process  Performance 
Sub-Factor 


2.1 

Resources  and  Cost 
Sub-Factor 


A 


Process 


Process  Efficiency 
Process  Effectiveness 


Financial 
Performance 

Environment  & 
Support 


Product 


Process  Compliance  ► 

◄  < 


Personnel  Effort  ► 


Cost — 

Schedule  ► 

System  Performance  ► 

Process  Performance  ► 

Product  Assurance  Level  ► 
Technology 

Effectiveness 

Constraints 

Sub-Factor 

2.6 


Maintainability 

Functional 
Correctness 


4 


Milestone 

Completion 

Critical  Path 
Performance 

Work 

Progress 


Product  Quality 
Sub-Factor 
2.5 


Evolutionary 

Capability 


Schedule  and 
Progress 
Sub-Factor 
2.4 


Acquisition 
-►  Program  Office 
Influences 
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Acquisition  Program  Office  Subfactors 


*  Resources  and  Cost  -  Included  are  the  program-specific  personnel 
resources  responsible  for  performing  the  work  within  this  factor  and  the  Acquisition 
Program  Office  infrastructure,  facilities  and  other  resources  required  to  execute  the 
program 

*  Process  Performance  -  processes  used  by  the  organizations  and 

functions  in  the  Acquisition  Program  Office  factor  to  perform  the  work  and  produce  the 
products  for  which  they  are  responsible 

*  Size  and  Stability  -  addresses  the  size  and  stability  of  the  processes  used 
and  of  the  products  produced  by  the  organizations  and  functions  in  the  Acquisition 
Program  Office  factor 

*  Schedule  and  Progress  -  addresses  the  schedule  and  progress  of  the 
work  performed  by  the  organizations  and  functions  in  the  Acquisition  Program  Office 
factor 

*  Product  Quality  -  addresses  the  quality  of  the  products  produced  by  the 
organizations  and  functions  in  the  Acquisition  Program  Office  factor 

*  Constraints  -  addresses  the  various  constraints  imposed  by  the  organizations 
and  functions  in  the  Acquisition  Program  Office  factor  on  all  types  of  work  (engineering, 
management,  support)  being  performed  by  the  contractor  on  the  program 
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Acquisition  Program  Office  Influences  and 

Example  Metrics  - 1 


Subfactor 

Influence  (Issue) 

Metric 

Resources  &  Cost 

Personnel  Effort 

Staff  Level 

Effort 

Staff  Turnover 

Overload 

Unplanned  Tasking 

Financial  Performance 

Budget 

Cost 

Environment  &  Support 

Quantity  Needed/Available 

Time  Needed/Available 

Fidelity  &  Quality  of  Resources 

Time  Used 
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Acquisition  Program  Office  Influences  and 

Example  Metrics  -  2 


Subfactor 

Influence  (Issue) 

Metric 

Process  Performance 

Process  Compliance 

Reference  Maturity 

Profile 

Process  Audit  Findings 

Process  Efficiency 

Productivity 

Cycle  Time 

Process  Effectiveness 

Defects  Contained 

Defects  Escaped 

Rework  Effort 

Rework  Components 

Size  and  Stability 

Processes 

Process  Size 

Process  Changes 

Products 

Product  Size 

Product  Changes 
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Acquisition  Program  Office  Influences  and 

Example  Metrics  -  3 


Subfactor 

Influence  (Issue) 

Metric 

Schedule  and 

Progress 

Milestone  Completion 

Milestone  Dates 

Schedule  Compression  Ration 

Critical  Path  Performance 

Slack  Time 

Work  Progress 

Requirements  Traced/Untraced 

Action  Items  Opened/Closed 

Change  Requests  Opened/Closed 

Evolutionary  Capability 

Capabilities  Fielded 

Interfaces  Integrated 

Product  Quality 

Maintainability 

Complexity 

Functional  Correctness 

Defects 

Age  of  Defects 

Completeness 

Estimating  Accuracy 
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Interaction  among  Acquisition  Program  Office 

Subfactors  and  Influences 


Size  &  Stability  Resources  &  Cost 


Product  Quality 
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Acquisition  Program  Office  Metrics: 
Cost  and  Schedule  Performance  Indexes 


Compare  actual  work  accomplished  on  a  project  to  the  original  plan, 
and  track  cost  and  schedule  performance  for  the  work  completed 


•  ANALYSIS  MODEL - 

>  CPI  and  SPI  are  plotted  over  time  to 
show  trends. 

>  Indices  greater  than  1 .0  indicate  under 
cost  (CPI)  and  ahead  of  schedule  (SPI). 

>  Indices  less  than  1.0  indicate  over  cost 
(CPI)  and  behind  schedule  (SPI). 

•  DECISION  CRITERIA - 

>  CPI  and  SPI  at  the  cumulative  level  are 
an  indication  of  the  health  of  the 
program. 

>  Values  less  than  0.9  are  an  indication 
that  there  are  problems  that  should  be 
addressed. 

>  The  overall  trends  should  be  reviewed. 
Several  months  of  declining  values  are 
an  indicator  that  there  are  issues  that 
should  be  addressed. 


SPO  SPI  and  CPI 


Year 

2001 

Mo  nth 

Jan  '01 

FebOl  ;  MarOl 

Apr'01 

May  01 

JunOl  Jul'01 

Aug  '01 

Sep  01  Oct'01 

No  vO  1 

Dec '01 

Activity 

Cumulative  SPI 

0.99  0.99 

0.98 

0.98 

0.98  0.98 

0.98 

0.99  0.99 

0.99 

0.99 

Monthly  SPI 

1.00  0.90 

0.61 

0.76 

0.76  0.57 

0.75 

0.93  0.93 

1.00 

1.00 

Cumulative  CPI 

1.00  1.00 

1.00 

1.00 

1 .00  0.99 

0.99 

0.97  0.97 

0.97 

0.97 

Monthly  CPI 

1.13  126 

0.97 

1.15 

0.96  0.82 

0.82 

0.69  0.69 

0.70 

0.70 

Cumulative  Schedule  Performance 
Index  (SPI) 


Cumulative  Cost  Performance 
Index  (CPI) 


Jan  Feb  Mar  Apr 
'01  ’01  01  01 


Framework 

Factor:  Acquisition  Program  Office 
Subfactor:  Resources  and  Cost 
Influence:  Financial  Performance 
Metric:  Cost 
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Acquisition  Program  Office  Metric: 
Unplanned  Tasking  Effort 

Compare  actual  effort  against  the  original  plan,  and  track  variance 

due  to  unplanned  tasking 

SPO  Effort 


•  ANALYSIS  MODEL - 

>  Cumulative  effort  (planned  staff-hours, 
actual  staff-hours  on  planned  work  and 
on  unplanned  tasking)  is  plotted  over 
time  to  show  trends. 

>  The  actual  effort  spent  on  unplanned 
tasking  should  stay  within  the  decision 
criteria. 

•  DECISION  CRITERIA - 

>  Effort  at  the  cumulative  level  is  an 
indication  of  the  health  of  the  program. 

>  More  than  10%  of  planned  effort  spent 
on  unplanned  tasking  indicates  a 
dilution  of  effort  on  planned  program 
tasks. 

>  The  overall  trends  should  be  reviewed. 
An  adverse  trend  over  3  months  is  an 
indicator  that  there  are  issues  that 
should  be  addressed. 
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Framework 

Factor:  Acquisition  Program  Office 
Subfactor:  Resources  and  Cost 
Influence:  Personnel  Effort 
Metric:  Unplanned  Tasking 
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Acquisition  Program  Office  Metric: 
Action  Items  Opened/Closed 


Understand  the  magnitude  of  the  impact  of  the  SPO  work  backlog 

by  tracking  action  item  status 


•  ANALYSIS  MODEL  - 

>  Action  items  are  plotted  over  time  to  show 
trends 

>  Bar  chart  shows  the  magnitude  of  the  action 
items  opened  and  closed  each  week;  line 
chart  indicates  the  cumulative  open  and 
closed  action  items  from  week  to  week 

>  Action  item  closure  rates  should  remain 
consistent  over  course  of  program  in  order  to 
prevent  SPO  communication  backlog 

•  DECISION  CRITERIA - 

>  Open  action  item  buildup  is  an  indication  of 
management  overload,  leading  to  a  lack  of 
communication  with  the  contractor. 

>  A  closure  rate  of  less  than  85%  is  generally 
an  indicator  of  SPO  overload. 

>  Consistent  information  exchange  backlog 
leads  to  continual  discovery  and  rediscovery 
of  the  same  issues. 


week  1  week  2  week  3  week  4  week  5  week  6  week  7  week  8 


Framework 

Factor:  Acquisition  Program  Office 
Subfactor:  Schedule  and  Progress 
Influence:  Work  Progress 
Metric:  Action  Items  Open/Closed 
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The  Reality  of  Contracting  Constraints 


Unification 


Acquisition 

Environment 


Constraints 


Acquisition - 

Program  - 1 

Office  ! 


Contractor 

Environment 


Constraints 


T  i  i 

Product 

Development 
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Acquisition  Program  Office  Constraints 

Subfactor  - 1 


Influence  (Issue) 

Example  Metrics 

Cost 

Contract  Dollars  Profile 

Fiscal  Year  Dollar  Profile 

Volatility 

Schedule 

Fixed  Milestone  Dates 

Volatility 

System  Performance 

Timing 

Accuracy 

Reliability/ Availability 

Maintainability 

Program  Specific  QPRs 

Volatility 

Process  Performance 

CMM/CMMI  Goals 

Processes 

Volatility 
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Acquisition  Program  Office  Constraints 

Subfactor  -  2 


Influence  (Issue) 

Example  Metrics 

Product  Assurance  Level 

Safety 

Security 

IV&V 

Quality 

Volatility 

Technology  Effectiveness 

Suitability 

Readiness  (TRL) 

Volatility 
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Selecting  a  Balanced  Set  of 
Acquisition  Program  Office  Metrics 


•  Balance  the  metrics  across  the  subfactors 

❖  Resources  and  Cost 

❖  Process  Performance 

❖  Size  and  Stability 

❖  Schedule  and  Progress 

❖  Product  Quality 

❖  Constraints 

•  Focus  on: 

❖  Areas  of  highest  risk,  and 

❖  Areas  where  meeting  process  improvement  goals  will  provide  the 
greatest  benefit 

•  Keep  the  number  of  Acquisition  Program  Office  metrics 
small,  especially  initially 

❖  Target  7  to  1 1  metrics  at  most 

•  Techniques  used  for  selecting  development  metrics  are 
useful  here  as  well  (e.g.,  GQM,  PSM) 
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Example:  Selecting  a  Balanced  Set  of 
Acquisition  Program  Office  Metrics 


•  Metrics  for  managing  the  work  of  the  Acquisition  Program  Office 

1 .  How  is  unplanned  tasking  affecting  the  program  work?  (Resources  and  Cost) 

2.  Is  the  actual  cost  of  the  Acquisition  Program  Office  work  tracking  to  the  plan? 
(Resources  and  Cost) 

3.  What  is  the  cycle  time  for  responding  to  contractor  requests  for  information? 
(Process  Performance) 

4.  How  stable  is  the  system  requirements  specification?  (Size  and  Stability) 

5.  What  is  the  critical  path  for  the  Acquisition  Program  Office’s  integration  tasks? 
(Schedule  and  Progress) 

6.  Is  there  a  growing  backlog  of  action  items  for  which  the  Acquisition  Program 
Office  is  responsible?  (Schedule  and  Progress) 

7.  How  many  defects  have  been  identified  in  the  released  system  requirements 
specification?  (Product  Quality) 

•  Metrics  for  understanding  the  driving  constraints  being  placed  on  the 
contractor 

8.  What  is  the  readiness  level  of  the  critical  technology  being  imposed? 

9.  How  volatile  are  the  critical  system  performance  parameters? 
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Conclusions 


•  This  research  effort  has  defined  an  integrated  framework  for 
software  acquisition  early  warning  indicators 

❖  Includes  all  participants  in  the  acquisition  and  development  environment, 
not  just  the  contractor’s  product  development  team 

❖  Addresses  the  full  program  life  cycle 

❖  Aids  in  understanding  the  impact  on  mission  success  of  all  of  the  factors 

❖  Aids  in  identifying  where  corrective  action  is  needed 

•  The  framework  provides  a  specific  measurement  perspective  for 
the  work  of  the  Acquisition  Program  Office 

❖  Provides  measures  for  the  influences  of  the  Acquisition  Program  Office, 
including  its  product  and  process  performance 

❖  Provides  a  basis  for  measurement  in  an  Acquisition  Program  Office 
process  improvement  effort 
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Acronyms 
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CARD  Cost  Analysis  Requirements  Document 

CDRL  Contract  Requirements  Data  List 

CPI  Cost  Performance  Index 

DSS  Draft  System  Specification 

FFRDC  Federally  Funded  Research  and  Development  Center 

GFE  Government  Furnished  Equipment 

GFI  Government  Furnished  Information 

GOTS  Government  Off  the  Shelf 

GQM  Goal-Question-Metric 

ICD  Interface  Control  Document 

IPS  Integrated  Program  Summary 

IRS  Interface  Requirements  Specification 

KDP  Key  Decision  Point 

MOIE  Mission  Oriented  Investigation  and  Experimentation 

QPR  Quantitative  Performance  Requirement 

PSM  Practical  Software  and  System  Measurement 

RFP  Request  for  Proposal 

SETA  Systems  Engineering  and  Technical  Assistance 

SMC  Space  and  Missile  Systems  Center 

SPI  Schedule  Performance  Index 

SPO  System  Program  Office 

SPS  System  Performance  Specification 

T&E  Test  and  Evaluation 

TEMP  Test  and  Evaluation  Master  Plan 

TRD  Technical  Requirements  Document 

TRL  Technology  Readiness  Level 

USAF  United  States  Air  Force 
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